Some storage systems, such as solid-state drives (SSDs), contain a non-volatile memory that is organized by physical addresses. A host uses logical addresses to read data from and/or write data to the storage system, and the storage system stores a logical-to-physical address map that the controller uses to translate a logical address from the host into a physical address of the non-volatile memory. The logical-to-physical address map is typically stored in the non-volatile memory; however, it is faster to access the logical-to-physical address map if it is stored in the storage system's volatile memory. In many situations, the storage system's volatile memory is not large enough to store the entire logical-to-physical address map, so portions of the logical-to-physical address map that are mostly-likely needed are read out of the slower non-volatile memory and into the faster volatile memory. Over time, portions of the logical-to-physical address map are swapped between the volatile and non-volatile memories.